Probabilistic Synchronous Parallel
نویسندگان
چکیده
Most machine learning and deep neural network algorithms rely on certain iterative algorithms to optimise their utility/cost functions, e.g. Stochastic Gradient Descent (SGD). In distributed learning, the networked nodes have to work collaboratively to update the model parameters, and the way how they proceed is referred to as synchronous parallel design (or barrier control). Synchronous parallel protocol is practically the building block of all distributed learning frameworks, and its design has direct impact on the performance and scalability of the system. In this paper, we propose a new barrier control technique Probabilistic Synchronous Parallel (PSP). Comparing to the previous Bulk Synchronous Parallel (BSP), Stale Synchronous Parallel (SSP), and (Asynchronous Parallel) ASP, the proposed solution effectively improves both the convergence speed and the scalability of the SGD algorithm by introducing a sampling primitive into the system. Moreover, we also show that the sampling primitive can be composed with the existing barrier control mechanisms to derive fully distributed PSP-based synchronous parallel. We not only provide a thorough theoretical analysis1 on the convergence guarantee of PSP-based SGD algorithm, but also implement a full-featured distributed learning framework called Actor System and perform intensive evaluation atop of it.
منابع مشابه
Application of Radial Basis Neural Networks in Fault Diagnosis of Synchronous Generator
This paper presents the application of radial basis neural networks to the development of a novel method for the condition monitoring and fault diagnosis of synchronous generators. In the proposed scheme, flux linkage analysis is used to reach a decision. Probabilistic neural network (PNN) and discrete wavelet transform (DWT) are used in design of fault diagnosis system. PNN as main part of thi...
متن کاملLearning Probabilistic Synchronous CFGs for Phrase-Based Translation
Probabilistic phrase-based synchronous grammars are now considered promising devices for statistical machine translation because they can express reordering phenomena between pairs of languages. Learning these hierarchical, probabilistic devices from parallel corpora constitutes a major challenge, because of multiple latent model variables as well as the risk of data overfitting. This paper pre...
متن کاملMachine Translation Using Probabilistic Synchronous Dependency Insertion Grammars
Syntax-based statistical machine translation (MT) aims at applying statistical models to structured data. In this paper, we present a syntax-based statistical machine translation system based on a probabilistic synchronous dependency insertion grammar. Synchronous dependency insertion grammars are a version of synchronous grammars defined on dependency trees. We first introduce our approach to ...
متن کاملOn Generative Parallel Composition 1
A major reason for studying probabilistic processes is to establish a link between a formal model for describing functional system behaviour and a stochastic process. Compositionality is an essential ingredient for specifying systems. Parallel composition in a probabilistic setting is complicated since it gives rise to non-determinism, for instance due to interleaving of independent autonomous ...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
عنوان ژورنال:
- CoRR
دوره abs/1709.07772 شماره
صفحات -
تاریخ انتشار 2017